Journal of Learning Design 


DESIGNING FOR EFFECTIVE LEARNING 


MEASURING COURSE LEARNING OUTCOMES 


Mohsen Keshavarz 
M.A. Educational Planning 
University of Tehran, Iran 
keshavarz_mohsen@yahoo.com 


Abstract 

Accreditation criteria of programs require effective learning outcomes, 
assessment with documented procedures, tools, results, and actions to close 
the assessment loop with broad faculty> involvement. This article describes a 
methodology’ for providing quantitative measurement of a course’s learning 
outcomes. The methodology’ uses a linkage matrix that associates each 
course learning outcome to one or more course assessment tool. The 
approach adopted provides a numeric score between 0 and 1 for each 
learning outcome with respect to each assessment tool and a combined score 
be calculated for each learning outcome from the tools associated with that 
outcome. The proposed methodology also provides insights into the 
consistency of the various assessment tools used to measure the achievement 
of a particular course learning outcome. The methodology’ described here 
has been successfully adopted in obtaining accreditation and reaccreditation 
to all programs offered by College of Computer Engineering at University’ of 
Tehran. 
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Introduction 

Academic program assessment and evaluation is becoming an important process in providing 
improved education to students through modified curriculum and instruction. Assessment has also 
become a tool of accountability in education by providing evidence on the effectiveness of 
teaching (Biney et ah, 2008). An assessment plan determines how well students are benefiting 
from a learning experience offered by a program of study. We accept here that assessment: 

... is an ongoing process aimed at understanding and improving student learning. It 
involves making our expectations explicit and public; setting appropriate criteria and high 
standards for learning quality; systematically gathering, analysing, and interpreting 
evidence to determine how well performance matches those expectations and standards; 
and using the resulting information to document, explain, and improve performance. 
When it is embedded effectively within larger institutional systems, assessment can help 
us focus our collective attention, examine our assumptions, and create a shared academic 
culture dedicated to assuring and improving the quality of higher education. 

(Angelo, 1995, pp. 7-9) 

Over the last few years, the learning outcomes of a subject or course of study have become the 
focus of national and international conferences, and in peer-reviewed education literature, as a 
means of assessing the knowledge and skills obtained from learning opportunities. Learning 
outcomes have applications at three distinct levels: (i) the local level of the individual higher 
education institution for course units/modules, programs of study and qualifications; (ii) the 
national level for qualifications frameworks and quality assurance systems; and (iii) 
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internationally (for wider recognition and transparency purposes (Adam, 2004). Learning 
outcomes focus on the measurable cognitive, behavioral and attitudinal development of students as 
they interact with a learning activity. They are what students are expected to demonstrate in terms 
of knowledge, skills, and attitudes upon completion of a learning experience (Adam, 2004; 
Ashiem, Gowan & Reichgelt, 2007). 

Learning outcomes and outcomes-based approaches have implications for curriculum design, 
teaching, learning and assessment, as well as quality assurance. They are likely to form an 
important part of the twenty-first century approaches to higher education and the reconsideration 
of such vital questions as to what, who, how, where and when we teach and assess (Adam, 2004). 
In terms of curriculum design and development, learning outcomes are at the forefront of 
educational change. They represent a change in emphasis from teaching to learning that 
characterise what is known as the adoption of a student-centered approach in contrast to traditional 
teacher-centered viewpoint. Student-centered learning produces a focus on the teaching-learning- 
assessment relationship and the fundamental links between the design, delivery and measurement 
of learning (Adam, 2004). 

To implement a learning outcomes approach, a program must first formulate program educational 
objectives (broad goals) that address institutional and program mission statements and are 
responsive to the expressed interests of program stakeholders. The program must then formulate a 
set of program learning outcomes (knowledge, skills, and attitudes the program graduates should 
have) that directly address the educational objectives and encompass certain specified outcomes 
that are related to the particular program being assessed. The program educational objectives and 
outcomes must be explicit. The next step is to formulate a set of measurable learning outcomes for 
each course in the curriculum. Based on these course learning outcomes, a mapping is constructed 
between the program learning outcomes and course learning outcomes. This mapping is used as a 
part of a process to provide a quantitative measurement of the attainment of program learning 
outcomes based on the degree to which course learning outcomes have been achieved according to 
specified criteria. 

Program learning outcomes are also assessed by using other indirect assessment tools such as 
alumni survey forms, exit survey forms, employer survey forms, and internships (Ashiem et ah, 
2007). Indirect measures include data from surveys of seniors and alumni, retention rates, 
graduation rates, and number of students progressing to advanced degrees. While they allow 
administrators, faculty, researchers, and consumers to infer the benefits to students from their 
years in college, they cannot report with precision exactly what students have learned or what they 
are capable of doing as a result of their university education. Historically, offices of institutional 
research and alumni offices have collected this kind of data. 

Direct measures provide more evidence of the increase in students’ knowledge and abilities over a 
period of time. Standardised tests, for example, the Collegiate Learning Assessment (CLA), 
provide one kind of direct measure. While the CLA assesses general education skills, other 
standardised tests can measure specific disciplinary knowledge. The Force Concept Inventory’, for 
instance, is used to determine students’ understanding of concepts in mechanics. 

Other examples of direct measures include assignments that ask students to perform some kind of 
conceptual task (e.g., create a concept map) or portfolios compiled over a course of study. It is 
important to emphasise that these student work products need to be systematically reviewed for 
evidence of learning in order for them to be of most use. For example, rubrics can be developed 
and used by groups of faculty or educational researchers to analyse papers, theses, or portfolios in 
order to assess learning. Grades, of course, can also be a measure of learning although how the 
grades are determined and reported can sometimes undermine their usefulness. 

However, course learning outcomes are crucial to the process. Among other things, they enable the 
program to demonstrate precisely how specific program learning outcomes are addressed in the 
curriculum. If course learning outcomes are then assessed continuously and the results are used to 
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improve instruction that address them, then the degree to which the program meets its self-selected 
goals must inevitably improve. The contribution of this article is to describe a methodology that 
can be used to provide a quantitative measurement of the attainment of each course learning 
outcome. 


Formulating Course Learning Outcomes 

Once the program goals and program learning outcomes have been articulated and the curriculum 
has been designed, measurable course learning outcomes must be developed for each course in the 
curriculum. Each course learning outcome must map to at least one program learning outcome to 
ensure that all courses in the program of study are addressing the overall program learning 
outcomes. This process also verifies whether each program learning outcome is addressed in at 
least one course (Laney College, 2007; University of Connecticut, 2007). Designing courses using 
learning outcomes leads to a more student-centred approach: it emphasises a shift from what staff 
members teach towards what the student is able to do on successful completion of the course. 
Specifically, learning outcomes can help staff focus on exactly what they want students to achieve 
in terms of both knowledge and skills; inform students of what is expected of them and help 
concentrate their efforts; and provide a useful guide to stakeholders about the general knowledge 
and understanding that a graduate possesses (Adam, 2004). 

A well-structured course should show clear alignment between the learning outcomes and the 
assessment criteria used on the course; in turn this leads to the design of appropriate assessment 
tasks, and to deliver the course in a way which enables students to reach the required outcomes. 
Biggs and Tang (2007) have developed the fundamental idea of constructive alignment, which is 
the process of synchronising teaching methods, learning activities, and assessment tasks with 
course’s learning outcomes. Alignment of each of these three elements with learning outcomes is 
crucial for effective teaching (Biggs & Tang, 2007; Warren, 2005). Teaching activities should be 
driven by course learning outcomes and should support students in their learning activities and 
prepare them for assessment. This alignment between learning outcome, learning and teaching 
methods, assessment tasks and assessment criteria makes the whole process transparent to the 
students and to other interested stakeholders. 

Course learning outcomes should specify the minimum acceptable (threshold level) standard for a 
student to be able to pass a course. This means that it is important to express learning outcomes in 
terms of the essential learning for a module or course, so a small number of learning outcomes of 
central importance should be developed rather than a large number of superficial outcomes. 
Learning outcomes should be written using action verbs so that students are able to demonstrate 
that they have learned or achieved the outcome (Reichgelt & Yaverbaum, 2002). 

In 1956, Benjamin Bloom headed a group of educational psychologists who identified three 
domains of educational activities (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956). These are: 
cognitive, affective, and psychomotor. Knowledge, understanding and intellectual skills fall under 
the cognitive domain. The affective domain refers to attitudes and the psychomotor domain covers 
manual and physical skills. The group further divided the cognitive domain into six levels that 
describe the learning process from the simplest to the most complex. These levels are: knowledge, 
comprehension, application, analysis, synthesis and evaluation. The first two of these relate 
specifically to knowledge and understanding, while the remaining four involve intellectual skills. 
While it might seem appropriate to concentrate on the lower two categories for lower level 
courses, it is recommended that students should be engaged in higher level activities on a smaller 
more focused scale from the outset (Scott, 2003). Bloom’s levels of cognitive skills are shown in 
Table 1 along with a description of each skill. 
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Table 1. Bloom’s cognitive levels (after Bloom et al., 1956) 

Category 

Description 

Knowledge 

recalling or remembering something without necessarily 
understanding, using, or changing it 

Comprehension 

understanding something that has been communicated without 
necessarily relating it to anything else 

Application 

using a general concept to solve problems in a particular situation; 
using learned material in new and concrete situations 

Analysis 

breaking something down into its parts; may focus on identification of 
parts or analysis of relationships between parts, or recognition of 
organizational principles 

Synthesis 

creating something new by putting parts of different ideas together to 
make a whole. 

Evaluation 

judging the value of material or methods as they might be applied in a 
particular situation; judging with the use of definite criteria 


The following is a list of verbs for use when creating student learning outcome statements (Scott, 

2003): 

• To measure knowledge (common terms, facts, principles, procedures), ask these kinds of 
questions: Define, Describe, Identify, Label, List, Match, Name, Outline, Reproduce, Select, 
State. Example: “ List the steps involved in building an information system.'” 

• To measure comprehension (understanding of facts and principles, interpretation of material), 
ask these kinds of questions: Convert, Defend, Distinguish, Estimate, Explain, Extend, 
Generalise, Give examples, Infer, Predict, Summarize. Example: “ Summarize the basic 
principles of software design. ” 

• To measure application (solving problems, applying concepts and principles to new 
situations), ask these kinds of questions: Demonstrate, Modify, Operate, Prepare, Produce, 
Relate, Show, Calculate, Solve, Use. Example: “ Calculate the shortest path from node A to 
node B in the following graph.” 

• To measure analysis (recognition of unstated assumptions or logical fallacies, ability to 
distinguish between facts and inferences), ask these kinds of questions: Diagram, 

Differentiate, Distinguish, Illustrate, Infer, Point out. Relate, Select, Separate, Subdivide. 
Example: “ Analyse the requirements of a school information system.” 

• To measure synthesis (integrate learning from differ oriented software design with structured 
softM’are design.” 

The following are guidelines assembled from various sources as well as the author’s experience in 

writing course learning outcomes: 

i. Action verbs from Bloom’s Taxonomy with an emphasis on higher-order thinking skills 
should be used. 

ii. To facilitate the assessing of outcomes, one verb per learning outcome should be used. 

iii. There should be between 4-8 learning outcomes for each course, in fact the fewer the 
better. 

iv. Course learning outcomes should describe what a student should be able to DO at the end 
of a course rather than what the instructor teaches. 

v. Course learning outcomes should be written in language that students (and those outside 
the field) are able to understand. 

vi. Course learning outcomes are typically not content-specific. 
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vii. Ideally, each course or program should include learning outcomes from more than one 
domain (cognitive, psychomotor, and affective). 

viii. Each course learning outcome should be measurable and can be assessed, preferably 
using more than one assessment tool. 

ix. Weak verbs such as ““be aware,” “appreciate,” “identify,” “read,” and “recognize,” are to 
be avoided in general. For example, recognising a phenomenon is weak compared to 
understanding that phenomena. 

Earlier courses in a program may have outcomes where students “explain” “describe,” and 
understand,” advanced courses should provide more analytical skills where students can “analyse” 
“design,” “implement ,” and “build” (Biney et ah, 2008; Warren, 2005). 

The following is an example of a poor learning outcome: “ Develop skills to analyse a large 
volume of data.” Obviously students should acquire these skills during the course and not develop 
them after finishing the course because that defeats the whole objectives of creating course 
learning outcomes. The following is an example of course learning outcomes written for a course 
of a program offered by the College of Computer Engineering at University of Tehran. After 
completing this course, students are able to: 

• explain the problem-solving process used to construct a computer program. 

• construct an algorithm using pseudo code. 

• select a suitable name, data type, and initial value for a variable or constant. 

• create code using selection control statements. 

• create code using repetition control statements. 

• manipulate data using arrays, strings and records. 

• use files for input and output processing. 

• construct and test a user defined function. 


Course Assessment Methods 

Assessment is usually classified into summative and formative for the purpose of considering 
different objectives of course assessment methods (Scriven, 1967). Summative assessment refers 
to the assessment of the learning and summarizes the achievements of learners at a particular point 
in time. After a period of work, the learner sits for a test and then the teacher marks the test and 
assigns a score. The test aims to summarize learning up to that point. Midterm and end of course 
exams fall into this category. In an educational setting, summative assessments are typically used 
for evaluation purposes to assign students a course grade. Formative assessment is generally 
carried out throughout a course or project. Formative assessment, also referred to as “educative 
assessment,” is used to aid learning. In an educational setting, formative assessment might be a 
teacher, peer, or the learner, providing feedback on a student’s work, and would not necessarily be 
used for grading purposes. Formative assessments are diagnostic. Robert Stake, an educational 
researcher, provided the following interesting analogy: When the cook tastes the soup, that’s 
formative; When the guests taste the soup, that’s summative (Stake, 1998). 

Summative and formative assessments are often referred to in a learning context as assessment of 
learning and assessment for learning respectively (Earl, 2003). Assessment of learning is 
generally summative in nature and intended to measure learning outcomes and report those 
outcomes to students, parents, and administrators. Assessment for learning is generally formative 
in nature and is used by teachers to consider approaches to teaching and next steps for individual 
learners and the class (Educational Assessment, 2006). 

In general, high-quality assessments are considered those with a high level of reliability and 
validity. Reliability relates to the consistency of an assessment method. A reliable assessment is 
one which consistently achieves the same results with the same or similar group of students. 
Reliability is affected by factors such as ambiguous questions, too many options within a question 
article, vague marking instructions and poorly trained markers. A valid assessment is one which 
measures what it is intended to measure. An exam is valid when it properly assesses the syllabus 
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upon which the examination is based. A common fonn of formative assessment is diagnostic 
assessment. Diagnostic assessment measures a student’s current knowledge and skills for the 
purpose of identifying a suitable program of learning. Self-assessment is a form of diagnostic 
assessment which involves students assessing themselves (Educational Assessment, 2006). 

Assessment methods should be designed such that they are able to measure the full range of 
outcomes associated with a particular course. For example, for information technology courses, 
they should include (Biney et al., 2008): 

• assessment methods that measure the ability of students in demonstrating subject 
knowledge, 

• designing and conducting experiments, 

• gathering data, analysing and interpreting data, 

• demonstrating and applying knowledge, 

• defining a technical problem, 

• planning a project, 

• conducting a review of the literature, 

• generating ideas and creativity, 

• perform preliminary and detailed design, 

• functioning effectively and as a member of a team, 

• solving technical problems, 

• defining computing requirements to solve a particular problem, 

• formulating and analysing engineering/technical/ computing problems, 

• solving engineering/technical/computing problems, 

• understanding and demonstration of ethical issues and professional responsibilities, 

• understanding and demonstration of social responsibilities, 

• written and oral communications, 

• making effective use of library and on-line resources, and 

• awareness of contemporary issues in industry. 

Current assessment tools used to assess courses learning outcomes in other countries include: Mid 
Term Exam, Final Written Exam, Short Article, Team Project, Oral Discussions, Lab work, 
Presentations, Seminars, Reports, Tests and Quizzes, Student Portfolio, and Individualised 
Products areas or solve problems by creative thinking, ask these kinds of questions: Categorise, 
Combine, Compile, Devise, Design, Explain, Generate, Organize, Plan, Rearrange, Reconstruct, 
Revise, Tell. Example: “Design a data flow diagram for the following software requirements 
specification:” To measure evaluation (judging and assessing), ask these kinds of questions: 
Appraise, Compare, Conclude, Contrast, Criticise, Describe, Discriminate, Explain, Justify, 
Interpret, Support. Example: “ Contrast object. ” 


Measuring Course Learning Outcomes 

The next step is to provide a data-backed quantitative measurement of how well students are 
achieving each course’s learning outcomes. The process used to get these measurements should be 
easy to implement and not time consuming to instructors. It is not acceptable to determine a 
student’s achievement of course learning outcomes on the basis of the final grade obtained in the 
course alone. These grades represent the aggregation of too many factors, causing the student’s 
ability in any particular topic area within the course to be lost in the aggregation (Lord, 1980). A 
more detailed level of analysis is needed. One approach that we suggest in this article is to create a 
linkage matrix that associates each course learning outcome with one or more assessment tool. 
This matrix is shown in Figure 1. 
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Figure 1. Linking course outcomes to assessment tools 


The measurement criteria is as follows: for each course outcome / and assessment tool J that 
address an outcome /, the maximum grade allocated for outcome I and the average scored grade 
obtained by students for outcome I is entered. An outcome 1 is achieved if the ratio 
total_scored/total_max. Where, total_scored is the sum of average grades obtained by students 
from all tools J for outcome /, and total jnax is the sum of the maximum grade of all tools used to 
measure outcome I. 

This approach of measuring the achievement of a particular course learning outcome can provide 
two very important observations. First, it indicates which course learning outcome the students 
have failed to achieve. Secondly, this measurement technique also indicate if a learning outcome 
has been achieved consistently by all assessment tools assigned to it which further provide an 
insight into the consistency of the various assessment tool in measuring a particular course 
learning outcome. 

The suggested methodology for providing a quantitative measurement of course learning outcomes 
may impose a heavy burden on the instructors in terms of designing the questions for each 
assessment tool and data collection and analysis if implemented manually. However, we feel that 
an automated system might alleviate much of the burden on instructors’ time and efforts. The 
assessment process described above applies successfully in obtaining accreditation and 
reaccreditation to all programs offered by College of Computer Engineering at University of 
Tehran. 


Closing The Assessment Loop 

Assessment of course learning outcomes is a continuous process and cyclical in nature. The 
assessment cycle as Implemented by College of Computer Engineering at University of Tehran is 
depicted in Figure 2. The final steps in the assessment cycle are referred to as “closing the 
assessment loop” (Bailie et al., 2010; Ramesh, & Mattiuzzi, 2001). Closing the loop refers to the 
process of using the results obtained from various assessment activities to improve the program 
and to document such improvements with the intent to positively impact future student learning. In 
Figure 2, the shaded rectangles represent the steps required to close the assessment loop. 
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Closing 


tlie 


Loop 



Figure 2. Closing the assessment loop 

After course instructors conduct assessment activities, collect assessment data and analyse it, the 
course coordinator provides the quantitative measurement regarding the achievement of each 
course learning outcome as described in the table shown in Figure 1. This information will be 
reviewed by the College Assessment Committee. In addition, the College Assessment Committee 
also takes into consideration the results of student’s surveys which are conducted for each course 
towards the end of the semester. These surveys provide valuable information with regard to the 
suitability of textbook and references; academic background and prerequisite courses; course 
delivery modes; and lab activities among others. 


Conclusion 

In this article, a description of the full assessment cycle for course learning outcomes and what 
actions are needed to close the assessment loop have been presented. The main contribution of the 
article is in proposing a methodology for providing a quantitative measurement of the level to 
which each course learning outcome has been achieved. In addition, this methodology provides 
valuable information regarding how each learning outcome is being assessed by the different 
assessment tools giving insights into the consistency of the various tools in measuring a particular 
course learning outcome. The approaches successfully apply in obtaining accreditation to all 
degree programs at the Colleges. 
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